Dynamic range extension by combining differently exposed hand-held device-acquired images

ABSTRACT

Two or more digital images of a same scene are captured with different exposure levels. Image pairs are generated that have a same exposure level from the image data of differently exposed images by adjusting the exposure of at least one of the pair of images. A ghosting map is generated based on differences between the pair of images adjusted to the same exposure. A blurred ghosting map is generated and alpha blending is performed.

PRIORITY

This application is a 371 of PCT Application Ser. No. PCT/IB2012/000381,filed Feb. 17, 2012, which claims priority to U.S. provisional patentapplication Ser. No. 61/444,565, filed Feb. 18, 2011.

BACKGROUND

Modern digital photo cameras have a limited dynamic range capability,usually not being enough to capture all the details in a scene. The usertherefore has to choose between a picture that has details in shadowsbut is overexposed and one that properly exposes highlights but isunderexposed everywhere else. Capturing multiple images of the samescene at different exposure levels and combining them is currently themost common way to achieve a high dynamic range photograph. This offersbest results for tripod mounted cameras and static scenes. Conventionaldisplay devices have a limited dynamic range and different tone-mappingtechniques may be applied to compress the dynamic range of the HDRimage, preserving certain relevant details.

Typically, combining two image captures with different exposures andcontent can tend to produce a high dynamic range (HDR) image withundesired ghosting artifacts, particularly when a handheld digital stillcamera or camera-phone is being used. Taking multiple exposures with asingle aperture tends to lead to ghosting artifacts caused by movingobjects in the scene between exposures. Usually by compressing thedynamic range using a tone-mapping technique the contrast of an imagemay be reduced, although some details are typically lost and the overallappearance intended by the camera's manufacturer is severely altered.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a workflow of a technique for extending a dynamicrange of a scene captured in a digital image in accordance with certainembodiments.

FIG. 1B illustrates a workflow of another technique for extending adynamic range of a scene captured in a digital image in accordance withcertain embodiments.

FIG. 2 illustrates a workflow of another technique for extending adynamic range of a scene captured in a digital image in accordance withcertain embodiments.

FIG. 3 schematically illustrates a workflow in accordance with anothertechnique for extending the dynamic range of the captured scene inaccordance with certain embodiments.

FIG. 4 illustrates an underexposed image.

FIG. 5 illustrates an overexposed image.

FIG. 6 illustrates a HDR image and/or image with extended dynamic rangegenerated by the combination of the underexposed and overexposed imagesadvantageously without ghosting artifacts in accordance with certainembodiments.

FIG. 7 illustrates a lightened underexposed picture.

FIG. 8 illustrates a darkened overexposed picture.

FIG. 9 illustrates a ghosting map in accordance with certainembodiments.

FIG. 10 illustrates a corrected underexposed picture in accordance withcertain embodiments.

FIGS. 11A and 11B illustrate a mask used for alpha blending inaccordance with certain embodiments.

FIG. 12 illustrates an image having an extended dynamic range and/or HDRpicture without ghosting advantageously generated by a technique inaccordance with certain embodiments.

FIG. 13 is a plot of alpha value versus an image data parameter for analpha blending process in accordance with certain embodiments.

FIG. 14 illustrates a global HDR or extended dynamic range image inaccordance with certain embodiments.

FIG. 15 illustrates a region-based HDR image or extended dynamic rangeimage in accordance with certain embodiments.

FIG. 16A-16F schematically illustrate an example of a technique forproducing a HDR image and/or for extending a dynamic range of an imagewithout ghosting artifacts in accordance with certain embodiments.

FIG. 17A-17E schematically illustrate an example of a technique for HDRimage blending in accordance with certain embodiments.

DETAILED DESCRIPTIONS OF THE EMBODIMENTS

Embodiments are provided including methods for generating an outputimage with an extended dynamic range. The method may include acquiringfirst and second digital images of approximately a same scene that havefirst and second exposure levels, respectively. An adjusted version ofthe first digital image may be generated as an adjusted first digitalimage. The exposure level of the first digital image may be globallymodified to approximately a same first matching exposure level as thesecond digital image. The second digital image may be globally alignedwith the first or adjusted first digital image or both. A ghosting mapmay be generated based on a first threshold and absolute differencesbetween corresponding pixels of the adjusted first digital image and thesecond digital image. One or more burnt regions of the second image maybe determined using a second threshold. A burnt pixel map may be createdof the burnt regions. A blurred ghosting map may be generated. Pixels ofthe ghosting map may be blurred that correspond to the burnt regions inthe burnt pixel map. An adjusted version of the second digital image maybe generated as an adjusted second digital image. An exposure level ofthe second digital image may be globally modified to approximately asame second matching exposure level as the first digital image. Themethod may include alpha blending the adjusted second digital image andthe first digital image based on the blurred ghosting map to generate asafe first digital image, as well as alpha blending the safe firstdigital image and the second digital image based on the burnt pixel mapto create an output image.

The generating of the blurred ghosting map may include applying theblurred burnt pixel map to pixels of the ghosting map after the blurringof the pixels of the ghosting map. The method may further includeblurring the burnt pixel map before alpha blending the safe firstdigital image and the second digital image.

The generating of the blurred ghosting map may include applying theblurred burnt pixel map to pixels of the ghosting map after the blurringof the pixels of the ghosting map. The burnt regions of the second imagemay contain less detail due to overexposure and saturation thancorresponding regions of the first image. The burnt regions of thesecond image may contain less detail due to underexposure and darknessthan corresponding regions of the first image. The ghosting map mayinclude a binary ghosting map.

The first matching exposure level may include the second exposure level,and the second matching exposure level may include the first exposurelevel. Alternatively, the method may include adjusting the exposurelevels of the first and second digital images to the second and firstmatching exposure levels, respectively, to match the exposure levels ofthe adjusted second and first images.

A digital image acquisition device is also provided that includes ahousing, a lens and image sensor within the housing configured forcapturing digital images, a processor, and a memory having storedtherein code for programming the processor to perform a method ofgenerating an output image of a scene with an extended dynamic range inaccordance with any of the methods described herein. The acquiring ofthe first and second digital images may or may not involve using thelens and the image sensor.

One or more non-transitory digital storage media are also providedhaving stored therein code for programming a processor-based and cameraenabled device to perform a method of generating an output image of ascene with an extended dynamic range in accordance with any of themethods described herein.

Embodiments are provided to produce HDR images without the undesiredghosting artifacts. The term HDR or high dynamic range is meant toinclude images that would meet a standard definition of a “HDR image,”such as an image having a greater dynamic range between lightest anddarkest areas than the digital image acquisition device that capturedthe image is mechanically and/or optically configured to provide and/orthat was set to provide when the image was captured. In otherembodiments, an HDR image has a greater dynamic range than aconventional digital image acquisition device (with average, minimal,optimal and/or a selected dynamic range) is mechanically or opticallyconfigured to provide. In other embodiments, a HDR image is generatedusing an advantageous method, digital storage media having code storedtherein for programming a processor, and/or a processor-based digitalimage acquisition device having code stored therein for programming thedevice to extend the dynamic range of an acquired and/or captureddigital image. A HDR image in accordance with further embodiments mayhave the same dynamic range, such as may be represented for example on 8bpc.

A HDR image in accordance with certain embodiments can be displayed onconventional display devices, while rendering a greater dynamic range ofthe scene being displayed. A device in accordance with certainembodiments may be described as being configured to capture an extendeddynamic range of a scene.

In certain embodiments, one of two pictures of approximately a samescene is transformed so that it has the same content as the other one,but with different exposure. More than two images may be used inalternative embodiments. Ghosting removal in accordance with certainembodiments is independent of the blending method used, i.e., having twoimages with the same content but different exposure, any of multipleimage blending methods may be used. In one example, alpha blending maybe used. By combining large regions of different images in accordancewith embodiments described herein, original images may be processed by acamera's ISP without using tone mapping, while not significantlyaltering the overall appearance intended by the producer and stillextending the dynamic range of the captured scene.

An example of a workflow in accordance with certain embodiments isillustrated at FIG. 1 a Two (or more) images of substantially orapproximately the same scene are acquired at step 1 at different levelsof exposure. These could be captured at or near the same time by a samesensor or different sensors, such as in successive, sequential orproximately captured images from an image acquisition device offeringspeeds of long exposure cameras for very dark scenes, to a few toseveral to 30, 60, 120 or more frames per second, and may be separatedby longer times proportional to the degree of stillness of the scene.When the different exposure levels of the two images is greater, thedynamic range can be typically more greatly extended using a techniquein accordance with certain embodiments. In certain embodiments, the twoimages include an underexposed (UE) image and an over-exposed (OE)image. The exact degree of over- or under-exposure will depend on theprecise acquisition conditions for the images. An example of anoverexposed image in accordance with certain embodiments would be animage having some degree of detail lost in 1%, 5%, 10%, 15%, 20%, 25% or30% or more of the over-exposed image due to saturation of one or morepixels, and an example of an underexposed image in accordance withcertain embodiments would be an image having some degree of detail lostin 1%, 5%, 10%, 15%, 20%, 25% or 30% or more of the under-exposed imagedue to darkness of one or more pixels. In one specific example, somedegree of detail may be lost in 15% to 25% of one or both images, or inother embodiments more selectively precise around a 1% to 4% range ofapproximately 20%. These characteristics may vary according to theacquisition conditions and the desired output parameters.

In one embodiment, under- and over-exposed images are captured at −2 eVand +2 eV exposure stops, while 0 eV would be the exposure setting of anormally exposed image or the preset or presumed or selected exposure ofa normal image around which the over- and under-exposed images may becaptured. In certain embodiments, if there are too many dark or lightareas in the under- or over-exposed image scene, respectively, then thenormally exposed image can be used as a substitute for the under- orover-exposed image. While this adjustment would reduce the HDR range ordegree of dynamic range extension provided by the technique, it wouldhave the advantage of enhanced assurance of producing a working outputfrom each HDR acquisition cycle. These counter-balancing parameters maybe selectively matched by the camera automatically, and/or using somemanual or acquired or measured input

In another embodiment, a normally exposed image may be captured andanalyzed to determine, calculate, predict or estimate how many exposurestops can be applied for the OE and UE images while maintaining areasonable or selected or predetermined assurance of some degree ofworking output. For example, if a larger area of an image scene has highluminance, then perhaps only +1 eV or +1.5 eV stops might be applied forthe OE image instead of the example provided above of +2 eV, and −1 eV,−1.5 eV, −2 eV, −2.5 eV, −3 eV or otherwise might be applied for theunderexposed image. These are just examples that apply to certain imagedata under certain acquisition conditions, and these levels may beadjusted or selected or predetermined based on scene analysis of one ormore preview or postview images, or based on a scene luminancemeasurement, or on analysis of another reference image or image data ormetadata, and may be performed automatically by the camera and/or basedon camera inputs and/or with or without some manual input

In some embodiments where the imaging device can perform exposuremeasurement across a scene, or where the camera has learned fromprevious image captures, or wherein the camera has been pre-programmedmanually or in response to measured or programmed inputs, thedetermination of these settings may be made without a first acquisitionof a normally exposed image.

As these two images are acquired at slightly different times there maybe a global movement of the camera between the two acquisitions and thusa means of globally aligning the two images is employed in accordancewith certain embodiments. An exemplary technique for global imagealignment is described in US 2008/0309769 to Albu et al., incorporatedby reference.

Referring to FIG. 1A, the two images are globally aligned in accordancewith certain embodiments before step 2. In alternative embodiments,alignment is performed after step 2. Step 2 involves adjusting theunder-exposed image to bring it to approximately the same exposure levelas the over-exposed image providing an “adjusted under-exposed” image(AUE). This can be achieved by stretching the image histogram, orotherwise. As the two images are now aligned and at similarluminance/exposure levels any substantial differences at the pixel levelbetween the two images are due to local motion, or potential “ghosting”between the two images.

The third step (step 3) of the example process illustrated at FIG. 1Ainvolves creating a ghosting map, e.g., a binary ghosting map, wherevariations, beyond a threshold, between the two aligned and exposurematched images represents a potential ghost pixel. Various techniquescan be employed including relative and adaptive thresholds in certainembodiments based on characteristics of one or both of the images, buteven a static threshold can be used in certain embodiments to constructthe initial binary ghosting map (BGM) which determines if a pixel islikely to be a ghost pixel or not. Typically this determination is madeon the luminance channel if available (e.g. in YCC color space), but mayoptionally be determined using multiple color channels and applying amajority voting system (e.g. in RGB color space with noticeablevariation on 2 of 3 channels to confirm a ghost pixel).

Now as mentioned previously some areas of the overexposed (OE) image maybe of such high luminance that they are effectively saturated andotherwise useful structural details or texture has been lost withinthese image regions. In certain embodiments, the existence of suchregions determines if an image is over-exposed or not, and the level ofexposure of the overexposed image can depend in certain embodiments onthe percentage or degree of existence of these regions and/or on thepercentage or degree of loss of detail due to saturation. Similar forthe underexposed image with regard to detail lost due to darkness ofpixels. These regions are referred to herein as burnt regions whetherthey involve detail lost due to saturation in the OE or darkness in theUE.

Step 4 involves determining burnt regions. This can be using a static ordynamic threshold, e.g. luminance pixels >240, or with moresophisticated image analysis techniques. The goal is to clearlydelineate regions where image detail is destroyed by overexposure. Theseregions are known as “burnt” image regions and the map of such pixels isreferred to as the burnt regions map (BRM) and like the ghosting map itis also in certain embodiments a binary map, such that the pixels areeach classified as burnt, or not.

Step 5 of the example process of FIG. 1A, involves blurring the binaryghosting map (BGM) within regions which overlap with the BRM. Thiscreates a grayscale map with values running from 0-255, from the binaryghosting map, known as a “blurred binary ghosting map” (BBGM). It alsoeliminates any sharp binary transitions between ghosting map pixels.

At step 6, basically the reverse of step 2 is performed includingadjusting the exposure of the OE image to that of the UE image usinghistogram matching, or equivalent methods. This new image is the“adjusted” output image (AOE) and after exposure adjustment it should bealmost the same as the UE image except for (i) ghost pixels and (ii)burnt pixels. Now in the burnt pixels regions as image structure andtexture was already lost there can be no recovery of image data withinthese regions. But as the AOE and the UE images are now at the sameexposure, these regions can be restored to the AOE image from the UEimage.

In certain embodiments, when the dynamic range is meant to be moderatelyor slightly extended, there may be fewer burnt pixels or no burnt pixelseither due to saturation for the OE or due to darkness for the UE. Inthose embodiments, the steps described to compensate for burnt regionsare either not included or optional. Thus, in an extreme case whereexposure levels for the OE and UE result in no burnt regions due tosaturation nor to darkness, an alternative embodiment includes a methodfor generating an output image with an extended dynamic range thatinvolves acquiring first and second digital images of approximately asame scene that have first and second exposure levels, respectively. Anadjusted version of the first digital image is generated as an adjustedfirst digital image. The exposure level of the first digital image isglobally modified to approximately a same first matching exposure levelas the second digital image. The second digital image is globallyaligned with the first or adjusted first digital image or both. Aghosting map is generated based on a first threshold and absolutedifferences between corresponding pixels of the adjusted first digitalimage and the second digital image. An adjusted version of the seconddigital image is generated as an adjusted second digital image. Anexposure level of the second digital image is globally modified toapproximately a same second matching exposure level as the first digitalimage. The method further includes alpha blending the adjusted seconddigital image and the first digital image based on the ghosting map togenerate an output image. Regions may be blurred for various reasons,and if so, then a blurred region map may be generated, and alphablending of an intermediate safe underexposed image may be blended withthe overexposed image based on this blurred region map to yield a finaloutput image.

Referring back now to FIG. 1A, at step 7, alpha blending of the UE andAOE images is performed based on the BBGM map. This effectively copiesimage data from the UE image into the burnt regions of the AOE image tocreate a “safe” OE image (SUE).

At step 8 the BRM map is also blurred to provide a grayscale BBRM. Thefinal step 9 depicted in FIG. 1A involves combining the SUE image withthe original OE image using the BBRM to alpha blend the two imagestogether into a final output image.

In an alternative embodiment, shown in FIG. 1B, the over exposed imagecan be adjusted down to the exposure level of the underexposed image toprovide an AOE image instead at step 2. A ghosting map is similarlydetermined for step 3, but it is regions of the UE image that are ofsuch low luminance that no image structure or texture can be discernedin such regions. These regions are analogous to burnt regions althoughthey are more correctly known as “dark” regions per step 4. Burntregions (actually “dark” regions) are determined and the ghosting map isblurred within such regions per step 5. In steps 6 & 7, an adjustedunder-exposed image (AUE) is brought to the same exposure as the OE andcombined using the BBGM to generate a safe overexposed (SOE) image.Finally the SOE and UE are combined using the BBRM to yield a finaloutput image.

In a further alternative embodiment illustrated in FIG. 2, the blurredBRM is generated just after the original BRM in step 4 and applied tothe blurred binary ghosting map during step 5 to provide a refinedblurred binary ghosting map (R-BBGM). Steps 6 & 7 remain the same, butthe R-BBGM is used in step 7 rather than the BBGM. The final stepremains the same, combining SUE and OE, alpha blended with BBRM togenerate the final output image. This variant can provide moreaesthetically pleasing results in certain conditions of imageacquisition.

Another example process in accordance with certain embodiments is asfollows. Two or more image frames are captured with different exposures.Automatic exposure calculations may be performed using preview frames.In this way, an entire scene dynamic range, i.e., shadows andhighlights, is captured. Next, the images are registered. Theregistration process compensates for camera motion during image captureand enables HDR capture with hand-held cameras even without use of atripod. Ghosting artifacts are caused by moving objects between frames.Ghosting correction is next performed in accordance with certainembodiments and as described below. Now, HDR image blending isperformed. An advantage technique is used for image blending inaccordance with certain embodiments which preserves global contrastwhile maintaining a natural look and the overall image aspect designedby the camera's manufacturer. Other classical HDR merging andtone-mapping methods can be safely applied without the risk of ghostingartifacts, using the registered images with the same structural content.

Referring now to FIG. 3, a first frame is acquired 102. In this example,the first frame or first image is under-exposed while in otherembodiments the first frame is over-exposed or normally exposed. Asecond frame or second image is acquired at 104. In this example, thesecond frame or second image is over-exposed while in other embodimentsthe second frame is under-exposed or normally exposed. FIGS. 4 and 5,respectively, illustrate under-exposed and over-exposed images. Whilethe two differently-exposed images of the approximately same scene mayvary in terms of their relative and/or absolute exposure levels, in signand/or magnitude, from the examples provided herein in accordance withembodiments, the technique will provide images with advantageouslyextended dynamic ranges at various different exposure values and/or stopsettings for each of the two images. The degree of the extending of thedynamic ranges will vary depending on the details of the relativemagnitude and/or sign of the difference between the two exposure levels.

The first and second images will not necessarily have precisely the samecontent. Thus, by combining images 102 and 104, ghosting will typicallyresult as illustrated in the example picture of FIG. 6. The first andsecond frames are registered 106, so that the system now has aregistered underexposed frame 108 and a registered overexposed frame110.

Now, a histogram-based exposure compensation block 112 is applied to theunderexposed and overexposed registered frames 108 and 110. Imagehistograms are analyzed to, in some embodiments, bring the overexposedregistered frame 110 into the same exposure space as the underexposedregistered frame 108 and, respectively, the underexposed registeredframe 108 into the exposure space of the overexposed registered frame110. This produces a registered, lightened underexposed frame 114 and aregistered, darkened overexposed frame 116. Examples of 114 and 116 areshown respectively at FIGS. 7 and 8.

Lightening the underexposed frame 108 to the same exposure level as theoverexposed frame 110, to generate the lightened underexposed image 114of FIG. 7, has the advantage that the lightened underexposed image 114is now at approximately the same exposure level as the overexposed frame110 without adjusting the overexposed frame 110 in terms of itsexposure. Likewise, darkening the overexposed frame 110 to the sameexposure level as the underexposed frame 108 to generate the darkenedoverexposed image 116 of FIG. 8, has the advantage that the darkenedoverexposed image 116 is now at approximately the same exposure level asthe underexposed frame 108 without adjusting the underexposed frame 108in terms of its exposure.

With regard to processing power and computational resources, asignificantly lower computational effort is generally involved inadjusting the exposure of one image compared with two (or two comparedwith four, etc). That is, simply put, the effort is about half to adjustone image instead of two. For a 16 Mp image, e.g., or perhaps even amore data rich image as may be available, this advantage can beappreciable, particularly when the computations are being performed onembedded devices, whether they be camera-enabled or not, or on miniatureor full-size digital cameras, or if network connections are involved,among other considerations.

Also, an overexposed image 110 can contain “burnt” regions, e.g., lossof detail due to saturation, and likewise an underexposed image 108 cancontain “burnt” regions, e.g., loss of detail due to excessive darkness,and adjusting the overexposed image 110 to generate image 116, and image108 to generate image 114, can tend to generate some false data withregard to the burnt regions. While compensation for these may beprovided in techniques provided in accordance with certain embodiments,e.g., by blurring a ghosting map and/or by generating and blurring aburnt region map, as described with reference to FIGS. 1A, 1B and 2,these false data may either not be fully controlled in the beginningphase that involves creating image pairs (108, 116) and (110, 114) atsame exposure levels from the image data of the image pair (108, 110)captured at different exposure levels, or may be substantiallycontrolled at some higher cost in terms of resources, introduction ofartifacts and/or a decrease of the global performance.

With these considerations in mind, nonetheless, in alternativeembodiments, image pairs may be created that are at an approximatelysame exposure level, yet are at another exposure level different fromeither of the exposure levels of the overexposed image 110 or theunderexposed image 108. In the example where those levels respectivelyare +2 eV and −2 eV, then image pairs may be created with each imagebeing at an approximately a same exposure level different from either ofthese two original levels, such as +1.5 eV, −1.5 eV, +1.9 eV, −1.9 eV,+1 eV, +0.5 eV, −1 eV, −0.5 eV. Even levels higher than +2 eV or lowerthan −2 eV, or whatever the original levels are, are possible and couldprovide advantage. However, most embodiments, as described, involveselecting levels for the overexposed frame 102 and the underexposedframe 104 that are far apart so that the dynamic range can be extendedfarther. In one embodiment, those levels are selected to be as far apartas possible without losing too much or an excessive or over the limitamount of detail due to saturation in the overexposed frame 102, 110and/or to darkening in the underexposed frame 104, 108

A ghosting detection block 118, as illustrated in FIG. 3, is thenapplied to registered overexposed frame 110 and registered, lightenedunderexposed frame 114. A ghosting map 120 is generated in certainembodiments and an example is illustrated in FIG. 9. A ghostingcorrection block 122 is applied to registered underexposed frame 108 andregistered, darkened overexposed frame 116 in the embodiment illustratedat FIG. 3. A registered underexposed corrected frame 124 is generatedand an example is illustrated at FIG. 10. An alpha blending block 126 isapplied to registered overexposed frame 110 and registered underexposedcorrected frame 124 as illustrated in FIG. 3. An example of a mask usedfor alpha blending 126 is illustrated in FIGS. 11A and 11B. Anadvantageous HDR image 28 is produced without ghosting as illustrated inthe example of FIG. 12.

Alpha blending generally involves a convex combination of two colorsallowing for transparency effects in computer graphics. The value ofalpha in the color code may range from 0.0 to 1.0, where 0.0 representsa fully transparent color, and 1.0 represents a fully opaque color.

The value of the resulting color when color Value₁ with an alpha valueof a is drawn over an opaque background of color Value₀ may be given by:Value=(1−α)Value₀+αValue₁

The alpha component may be used to blend the red, green and bluecomponents equally, as in 32-bit RGBA, or, alternatively, there may bethree alpha values specified corresponding to each of the primary colorsfor spectral color filtering. Similarly this can be applied to the YUVcolorspace equally on the luminance and on the chrominances, orindividually for each component.

Techniques in accordance with certain embodiments offer robust solutionsfor capturing HDR images and/or images having an extended dynamic rangeusing multiple exposures, i.e., two exposures in some embodiments oralternatively three or more exposures in other embodiments, and provideparticular advantage when working with non-static scenes and/or ahand-held or otherwise non-static camera. These techniques combineinformation from two different exposures, and work in certainembodiments according to the workflow illustrated at FIGS. 1A, 1B, 2 and3 and otherwise as described herein.

In another embodiment, use is made of low resolution camera preview (orpostview) frames, wherein a wide range of exposure values may beevaluated and the best two exposures chosen. The evaluation may be basedon one or more image histograms, and may include detecting a firstexposure 102 with correctly exposed highlights and a last exposure 104with well exposed shadows.

The two images (overexposed 104 and underexposed 102) are registered 106or aligned. Registration methods are described in the cited patentmatters below. For example, an integral projections-based registrationmethod may be used in certain embodiments. In certain embodiments, it isnoted that registration may alternatively or may also be performedbetween frames 108, 110 and/or frames 114, 116 and/or 110, 114 and/or108, 116 and/or between other frames that may involve differentprocesses or alternatives to embodiments of those described herein.

When the two frames are combined without compensation, ghostingartifacts caused by moving objects and image registration shortcomings(rotational movement or image distortion) can typically become apparent.In order to correct these artifacts, the two images are advantageouslybrought into the same domain in accordance with certain embodiments.Unlike other known methods that use the camera's response function totransform the two exposures, these embodiments utilize a histogram basedtransformation 112. The histogram of the underexposed image 108 may betransformed to resemble the histogram of the overexposed image 110, andvice-versa. This way, two new versions of the images are offered: thelightened underexposed image 114 and the darkened overexposed image 116.

Histogram-based exposure compensation in accordance with certainembodiments may involve the following. Two (or more) images are used asinputs in a HDR process or otherwise in extending a dynamic range,including a first underexposed image (e.g., −2 EV) and a secondoverexposed image (e.g., +2 EV). By the mentioned techniques, two newimages are produced: one is obtained by transforming the underexposedimage to match its histogram with the histogram of the overexposedimage, while the other one is obtained by transforming the overexposedimage to match its histogram with the histogram of the underexposedimage. The technique may in certain embodiments involve a tone-mappingtransformation that modifies an original image histogram to be more likethe histogram of the aimed image. The technique may be referred to as“Histogram Matching (Specification)” and a detailed description ofexamples of such technique can be found in Rafael C. Gonzalez andRichard E. Woods “Digital image processing”, Second Edition, Ed PrenticeHall, 2001, which is hereby incorporated by reference; and seeparticularly subsection 3.3.2, pages 94-103.

Using the overexposed image 110 and the lightened underexposed image114, a ghosting map 120 is computed using ghosting detection block 118.The two images 110,114 will have different structural content (caused bymoving objects or registration errors) but approximately the sameexposure level. By computing the difference between the two images withthe same exposure levels but different structural content and applying athreshold, a binary ghosting map 120 is obtained. To avoid steeptransitions between the ghosting affected areas and the rest of theimage, the ghosting map may be blurred in accordance with certainembodiments as indicated by BBGM in FIGS. 1A, 1B and R-BBGM in FIG. 2.In certain embodiments, the blurring is performed using a large Gaussiankernel. These kinds of transition are usually visible only in the lightareas of the images, so the blurring is applied in certain embodimentsto the regions where one of the image's luminance is greater than acertain threshold, while not being applied where the threshold is notmet.

Using the ghosting map 120 in the ghosting correction stage 122, theunderexposed image 108 is modified in certain embodiments by filling theghosting affected areas with content from the darkened overexposed image116. This way, the underexposed and overexposed images 108,116 willadvantageously have the same or more nearly the same structural contentand can be safely combined. This correction may be realized through analpha blending process between the underexposed image 108 and darkenedoverexposed image 116, using the previously obtained ghosting map 120 asalpha mask. The advantage of this approach is that compensation for thelocal motion identified in the underexposed image 108 is provided byreplacing it with a corresponding area from the overexposed image 116brought into the exposure space of the underexposed image 108.

The two images 108,116 are combined in certain embodiments by alphablending 126 using a mask. The mask may be computed by selecting whichareas are used from the overexposed image 116 according to the graphicpresented in FIG. 13 of the value of alpha ranging from 0 to 1 on they-axis and plotted versus an image data coordinate parameter. Blurringis applied in certain embodiments to avoid steep transitions. An image128 that captures an extended dynamic range of the scene is produced,for example, as illustrated in FIG. 14. The advantage of this method isthat large overexposed areas are selected from the underexposed imageand the underexposed areas from the overexposed image, and are mergedtogether without substantial changes preserving the overall imageappearance intended by the manufacturer.

In order to avoid possible ghosting artifacts, a further processing maybe applied on the ghosting map if the specific merging method presentedin the previous paragraph is used. Large areas are taken just from oneof the two input images, meaning no ghosting artifacts are possible inthose regions. Considering this, in certain embodiments, there is nocorrection applied for the underexposed image in regions where the alphablending mask is very close to 1 or 0. Therefore, the ghosting map maybe filtered and only values corresponding to the transition areasbetween images are kept.

Referring however to FIG. 15, the HDR or dynamic range extension effectmay be applied in certain embodiments on certain regions of interest, byusing information from a well exposed image for these regions. Theadvantage of this method is that by using information from only one ofthe exposures in sensible areas, the risk of any kind of artifacts isreduced, perhaps even to zero. Such areas can be faces or regionsinteractively selected by the user or by the device.

Ghosting removal is further illustrated at FIGS. 16A-16F. As mentioned,combining two image captures (or more) with different exposures andcontent can produce HDR images with ghosting artifacts as described andillustrated. FIG. 16A and FIG. 16B have different content, i.e., theperson walking has a different position of her head, arms and legs, forexample. The combination illustrated as FIG. 16C shows parts of bothimages, one being referred to as a translucent ghost of the other. Inaccordance with certain embodiments, however, one of the two pictures isadvantageously transformed so that it has the same content or some ofthe same content as the other one, but with different exposure. FIGS.16D and 16E show the person with head, arms and legs positioned in thesame places. Thus, FIG. 16F does not show the ghost artifacts that FIG.16C does.

Ghosting removal in accordance with certain embodiments may beindependent of blending method, as well. That is, any of a variety ofblending methods may be used to combine the two images which have thesame content but different exposure. In certain embodiments, HDR imageblending is performed as illustrated in FIGS. 17A-17E. Original LDR (lowdynamic range) images of FIGS. 17A and 17B, e.g, of +2 eV and −2 eV,respectively, are captured. A blending mask such as that shown in FIG.17C is used to produce the HDR image of FIG. 17D. Even the advantageousHDR image of FIG. 17E is possible in accordance with certainembodiments. In some embodiments, no region of interest is selected, andinstead a manual or automatic zoom-in feature may be used to confirm thesuccess of the technique or to otherwise more precisely determine aparameter or characteristic of an image before or after a certainprocessing is to be performed or is contemplated to be performed or hasbeen performed. In another example embodiment, a region of interest isselected just left of the central architectural structure of FIGS. 17Aand 17B. That is, features such as those flags hardly noticeable inFIGS. 17A and 17B are brought brilliantly in FIG. 17E using a region ofinterest feature in accordance with certain embodiments while theghosting artifacts that would have been present in a magnified region orinterest view have been eliminated because an extended dynamic rangeimage has been produced in accordance with certain embodiments withoutshowing ghosting artifacts either at all or within a tolerance dependingon the zoom range of the ROI feature and in any case thereby permittingthe image to have an extended dynamic range without being ruined in anylarge or small way by a ghosting artifact.

Advantageously in accordance with certain embodiments, a photograph thatcaptures a higher dynamic range of the scene is generated from twodifferently exposed low dynamic range pictures.

The use of the de-ghosting allows more permissive image acquisition. Thefact that artifacts are detected and corrected in accordance withcertain embodiments means that input images are used that differadvantageously from past solutions, preserving the overall image aspectas designed by the camera producer.

The use of images with greater exposure (eV) difference enablescapturing a larger dynamic range. Also, the use of images that differ incontent (camera translation, rotation, object moving) permit techniquesin accordance with certain embodiments to perform better for camerasheld in hand, and/or for larger duration between consecutiveacquisitions. Techniques in accordance with certain embodiments areadvantageously adapted for regions of interest. These may be faces, skinor anything that a user indicates or that may be pre-programmed in acamera-enabled device. For such a case, artifacts do not appear due touse of pixels from the same image, i.e., mixing is not used. The CRF andits inverse are generally not used in these embodiments. Exposurecompensation is done by histogram matching.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, except for those where a particularorder may be expressly set forth or where those of ordinary skill in theart may deem a particular order to be necessary.

In addition, all references cited above and below herein, as well as thebackground, invention summary, abstract and brief description of thedrawings, are all incorporated by reference into the detaileddescription of the preferred embodiments as disclosing alternativeembodiments.

The following belong to the same assignee and are hereby incorporated byreference for all purposes including as describing details of featuresand as disclosing alternative embodiments:

U.S. Pat. Nos. 7,620,218, 7,773,118, 7,660,478, 7,680,342, 7,692,696,7,551,755, 7,630,006, and 7,787,022; and

United States published patent applications nos. US2010/0329582,US2009/0303343, US2009/0179999, US2009/0167893, US2009/0179998,US2008/0309769, US2009/0263022, US2009/0080796, US2008/0219581,US2008/0309770, US2007/0296833, US2010/0026833, US2009/0304278,US2009/0185753, US2008/0316341, US2008/0219581, and US2008/0013798; and

U.S. patent application Ser. Nos. 12/959,281, 12/941,995, 12/907,921,12/941,983, 12/879,003, 12/636,647, 13/020,805, 61/406,970 and61/417,737.

What is claimed is:
 1. A digital image acquisition device, comprising: ahousing; a lens and image sensor within the housing configured forcapturing digital images; a processor; a memory having stored thereincode for programming the processor to perform a method of generating anoutput image of a scene with an extended dynamic range, wherein themethod comprises: acquiring first and second digital images ofapproximately a same scene at first and second exposure levels,respectively; generating an adjusted version of the first digital imageas an adjusted first digital image, including globally modifying theexposure level of the first digital image to approximately a same firstmatching exposure level as the second digital image; globally aligningthe second digital image with the first or adjusted first digital imageor both; generating a ghosting map based on a first threshold andabsolute differences between corresponding pixels of the adjusted firstdigital image and the second digital image; determining one or moreburnt regions of the second image using a second threshold; creating aburnt pixel map of the burnt regions; generating a blurred ghosting map,comprising blurring pixels of the ghosting map that correspond to theburnt regions in the burnt pixel map; generating an adjusted version ofthe second digital image as an adjusted second digital image, includingglobally modifying the exposure level of the second digital image toapproximately a same second matching exposure level as the first digitalimage; alpha blending the adjusted second digital image and the firstdigital image based on the blurred ghosting map to generate a safe firstdigital image; and alpha blending the safe first digital image and thesecond digital image based on the burnt pixel map to create an outputimage.
 2. The digital image acquisition device of claim 1, wherein thegenerating of the blurred ghosting map further comprises applying theblurred burnt pixel map to pixels of the ghosting map after the blurringof the pixels of the ghosting map.
 3. The digital image acquisitiondevice of claim 2, wherein the method further comprises blurring theburnt pixel map before alpha blending the safe first digital image andthe second digital image.
 4. The digital image acquisition device ofclaim 1, wherein the generating of the blurred ghosting map furthercomprises applying the blurred burnt pixel map to pixels of the ghostingmap after the blurring of the pixels of the ghosting map.
 5. The digitalimage acquisition device of claim 1, wherein the burnt regions of thesecond image contain less detail due to overexposure and saturation thancorresponding regions of the first image.
 6. The digital imageacquisition device of claim 1, wherein the burnt regions of the secondimage contain less detail due to underexposure and darkness thancorresponding regions of the first image.
 7. The digital imageacquisition device of claim 1, wherein the ghosting map comprises abinary ghosting map.
 8. The digital image acquisition device of claim 1,wherein the acquiring of the first and second digital images comprisesusing the lens and the image sensor.
 9. The digital image acquisitiondevice of claim 1, wherein the first matching exposure level comprisesthe second exposure level, and the second matching exposure levelcomprises the first exposure level.
 10. The digital image acquisitiondevice of claim 1, further comprising adjusting the exposure levels ofthe first and second digital images to the second and first matchingexposure levels, respectively, to match the exposure levels of theadjusted second and first images.
 11. One or more non-transitory digitalstorage media having stored therein code for programming aprocessor-based and camera enabled device to perform a method ofgenerating an output image of a scene with an extended dynamic range,wherein the method comprises: generating an adjusted version of a firstdigital image of a scene that was captured at a first exposure level asan adjusted first digital image, including globally modifying the firstexposure level to approximately a same first matching exposure level ofa second digital image that was captured of approximately a same scene;globally aligning the second digital image with one or both of the firstdigital image and the adjusted first digital image; generating aghosting map based on a first threshold and absolute differences betweencorresponding pixels of the adjusted first digital image and the seconddigital image; determining one or more burnt regions of the second imageusing a second threshold; creating a burnt pixel map of the burntregions; generating a blurred ghosting map, comprising blurring pixelsof the ghosting map that correspond to the burnt regions in the burntpixel map; generating an adjusted version of the second digital image asan adjusted second digital image, including globally modifying theexposure level of the second digital image to approximately a samesecond matching exposure level as the first digital image; alphablending the adjusted second digital image and the first digital imagebased on the blurred ghosting map to generate a safe first digitalimage; and alpha blending the safe first digital image and the seconddigital image based on the burnt pixel map to create an output image.12. The one or more non-transitory digital storage media of claim 11,wherein the generating of the blurred ghosting map further comprisesapplying the blurred burnt pixel map to pixels of the ghosting map afterthe blurring of the pixels of the ghosting map.
 13. The one or morenon-transitory digital storage media of claim 12, wherein the methodfurther comprises blurring the burnt pixel map before alpha blending thesafe first digital image and the second digital image.
 14. The one ormore non-transitory digital storage media of claim 11, wherein thegenerating of the blurred ghosting map further comprises applying theblurred burnt pixel map to pixels of the ghosting map after the blurringof the pixels of the ghosting map.
 15. The one or more non-transitorydigital storage media of claim 11, wherein the burnt regions of thesecond image contain less detail due to overexposure and saturation thancorresponding regions of the first image.
 16. The one or morenon-transitory digital storage media of claim 11, wherein the burntregions of the second image contain less detail due to underexposure anddarkness than corresponding regions of the first image.
 17. The one ormore non-transitory digital storage media of claim 11, wherein theghosting map comprises a binary ghosting map.
 18. The one or morenon-transitory digital storage media of claim 11, wherein the firstmatching exposure level comprises the second exposure level, and thesecond matching exposure level comprises the first exposure level. 19.The one or more non-transitory digital storage media of claim 11,wherein the method further comprises adjusting the exposure levels ofthe first and second digital images to the second and first matchingexposure levels, respectively, to match the exposure levels of theadjusted second and first images.
 20. A method of generating an outputimage with an extended dynamic range, comprising: acquiring first andsecond digital images of approximately a same scene that have first andsecond exposure levels, respectively; generating an adjusted version ofthe first digital image as an adjusted first digital image, includingglobally modifying the exposure level of the first digital image toapproximately a same first matching exposure level as the second digitalimage; globally aligning the second digital image with the first oradjusted first digital image or both; generating a ghosting map based ona first threshold and absolute differences between corresponding pixelsof the adjusted first digital image and the second digital image;determining one or more burnt regions of the second image using a secondthreshold; creating a burnt pixel map of the burnt regions; generating ablurred ghosting map, comprising blurring pixels of the ghosting mapthat correspond to the burnt regions in the burnt pixel map; generatingan adjusted version of the second digital image as an adjusted seconddigital image, including globally modifying an exposure level of thesecond digital image to approximately a same second matching exposurelevel as the first digital image; alpha blending the adjusted seconddigital image and the first digital image based on the blurred ghostingmap to generate a safe first digital image; and alpha blending the safefirst digital image and the second digital image based on the burntpixel map to create an output image.
 21. The method of claim 20, whereinthe generating of the blurred ghosting map further comprises applyingthe blurred burnt pixel map to pixels of the ghosting map after theblurring of the pixels of the ghosting map.
 22. The method of claim 20,further comprising blurring the burnt pixel map before alpha blendingthe safe first digital image and the second digital image.
 23. Themethod of claim 22, wherein the generating of the blurred ghosting mapfurther comprises applying the blurred burnt pixel map to pixels of theghosting map after the blurring of the pixels of the ghosting map. 24.The method of claim 20, wherein the burnt regions of the second imagecontain less detail due to overexposure and saturation thancorresponding regions of the first image.
 25. The method of claim 20,wherein the burnt regions of the second image contain less detail due tounderexposure and darkness than corresponding regions of the firstimage.
 26. The method of claim 20, wherein the ghosting map comprises abinary ghosting map.
 27. The method of claim 20, wherein the firstmatching exposure level comprises the second exposure level, and thesecond matching exposure level comprises the first exposure level. 28.The method of claim 20, further comprising adjusting the exposure levelsof the first and second digital images to the second and first matchingexposure levels, respectively, to match the exposure levels of theadjusted second and first images.